{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2019 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cascade (HD-CNN Model Deriative)\n",
    "\n",
    "## Objective\n",
    "\n",
    "This notebook demonstrates building a hierachical image classifer based on a HD-CNN deriative which uses cascading classifers to predict the class of a label from a coarse to finer classes. \n",
    "\n",
    "In this demonstration, we have two classes in the heirarchy: fruits and varieties of fruit. The model will first predict the coarse class (type of fruit) and then within that class of fruit, the variety. For example, if given an image of Apple Granny Smith, it would first predict 'Apple' (fruit) and then predict the 'Apple Granny Smith'.\n",
    "\n",
    "This deriative of the HD-CNN is designed to demonstrate both the methodology of heirarchical classification, as well as design improvements not available at the time (2014) when the model was first published [Zhicheng Yan](https://arxiv.org/abs/1410.0736).\n",
    "\n",
    "### General Approach\n",
    "\n",
    "Our HD-CNN deriative archirecture consists of:\n",
    "\n",
    "    1. An stem convolutional block.\n",
    "        - The output from the stem convolutional head is shared with the coarse and finer classifiers \n",
    "        (referred to as the shared layers in the paper).\n",
    "    2. A coarse classifier.\n",
    "        - A Convolution and Dense layers for classifying the coarse level class. \n",
    "    3. A set of finer classifiers, one per coarse level class.\n",
    "        - A Convolution and Dense layers per coarse level class for classifying the corresponding finer \n",
    "        level class.\n",
    "    4. A conditional execution step for predicting a specific finer classifier based on the output of the \n",
    "       coarse classifier.\n",
    "        - The coarse level classifier is predicted.\n",
    "        - The index of the prediction is used to select a finer classifier.\n",
    "        - An im-memory copy of the shared bottleneck layer (i.e., last convolution layer in stem) is passed as the\n",
    "          input to the finer level classifier.\n",
    "    \n",
    "\n",
    "Our HD-CNN deriative is trained as follows:\n",
    "\n",
    "    1. Train the coarse level classifier using the coarse level labels in the dataset.\n",
    "    \n",
    "<img src='arch-1.png'>\n",
    "    \n",
    "    2. Train the finer level classifier per coarse level class, using the corresponding subset (with finer\n",
    "       labels) from the dataset.\n",
    "       \n",
    "<img src='arch-2.png'>\n",
    "\n",
    "<br/>      \n",
    "## Dataset\n",
    "\n",
    "We will be using the Fruits-360 dataset, which was formerly a Kaggle competition. It consists of images of fruit labeled by fruit type and the variety. \n",
    "\n",
    "    1. There are a total of 47 types of fruit (e.g., Apple, Orange, Pear, etc) and 81 varieties.\n",
    "    2. On average, there are 656 images per variety.\n",
    "    3. Each image is 128x128 RGB.\n",
    "    \n",
    "<div>\n",
    "<img src='Training/Apple/Apple Golden 2/0_100.jpg' style='float: left'>\n",
    "<img src='Training/Apple/Apple Red 1/0_100.jpg'  style='float: left'>\n",
    "<img src='Training/Apple/Apple Red 1/0_100.jpg' style='float: left'>\n",
    "<img src='Training/Orange/Orange/0_100.jpg' style = 'float: left'>\n",
    "<img src='Training/Pear/Pear/0_100.jpg' style = 'float: left'>\n",
    "</div>\n",
    "\n",
    "\n",
    "## Objective\n",
    "\n",
    "The objective is to train a hierarchical image classifier (coarse and then finer label) using a cascading layer architecture. First, the shared layers and coarse classifier are trained. Then the cascading finer classifiers are trained. \n",
    "\n",
    "For prediction, the outcome (softmax) of the coarse classifier will conditionally execute the corresponding finer classifier and reuse the feature maps from the shared layers.\n",
    "\n",
    "## Costs\n",
    "\n",
    "This notebook requires 17GB of memory. It will not run on a Standard TF JaaS instance (15GB). You will need to select an instance with memory > 17GB."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "Download the Fruits 360 dataset from GCS public bucket into this JaaS instance.\n",
    "\n",
    "Some of the cells in the notebook display images. The images will not appear until the cell for copying the training data/misc from GCS into the JaaS instance is executed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying gs://cloud-samples-data/air/fruits360/fruits360-combined.zip...\n",
      "==> NOTE: You are downloading one or more large file(s), which would            \n",
      "run significantly faster if you enabled sliced object downloads. This\n",
      "feature is enabled by default but requires that compiled crcmod be\n",
      "installed (see \"gsutil help crcmod\").\n",
      "\n",
      "/ [1 files][230.9 MiB/230.9 MiB]                                                \n",
      "Operation completed over 1 objects/230.9 MiB.                                    \n",
      "arch-1.png\t\tmodel-coarse.h5    model-finer-1.h5  model-finer-8.h5\n",
      "arch-2.png\t\tmodel-finer-0.h5   model-finer-2.h5  model-finer-9.h5\n",
      "cascade-gap.ipynb\tmodel-finer-10.h5  model-finer-3.h5  model.h5\n",
      "cascade.ipynb\t\tmodel-finer-11.h5  model-finer-4.h5  Save\n",
      "fruits360-Cascade.zip\tmodel-finer-12.h5  model-finer-5.h5  Training\n",
      "fruits360-combined.zip\tmodel-finer-13.h5  model-finer-6.h5\n",
      "fruits.h5.h5\t\tmodel-finer-14.h5  model-finer-7.h5\n"
     ]
    }
   ],
   "source": [
    "!gsutil cp gs://cloud-samples-data/air/fruits360/fruits360-combined.zip .\n",
    "!ls\n",
    "!unzip -qn fruits360-combined.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started\n",
    "\n",
    "We will be using the fully frameworks and Python modules:\n",
    "\n",
    "    1. Keras framework for building and training models.\n",
    "    2. Keras builtin models (resnet50).\n",
    "    3. Keras preprocessing for feeding and augmenting the dataset during training.\n",
    "    4. Gap data engineering framework for preprocessing the image data.\n",
    "    5. Numpy for general image/matrix manipulation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from keras.applications.resnet50 import ResNet50\n",
    "from keras.preprocessing import image\n",
    "from keras.applications.resnet50 import preprocess_input, decode_predictions\n",
    "from keras.preprocessing.image import ImageDataGenerator\n",
    "from keras.layers import GlobalAveragePooling2D, Dense\n",
    "from keras import Sequential, Model, Input\n",
    "from keras.layers import Conv2D, Flatten, MaxPooling2D, Dense, Dropout, BatchNormalization, ReLU\n",
    "from keras import Model, optimizers\n",
    "from keras.models import load_model\n",
    "from keras.utils import to_categorical\n",
    "import keras.layers as layers\n",
    "from sklearn.model_selection import train_test_split\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import cv2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make Datasets\n",
    "\n",
    "### Make Coarse Category Dataset\n",
    "\n",
    "This makes the by fruit type dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Fruits(root):\n",
    "    n_label = 0\n",
    "    images = []\n",
    "    labels = []\n",
    "    classes = {}\n",
    "    \n",
    "    os.chdir(root)\n",
    "    classes_ = os.scandir('./')\n",
    "    for class_ in classes_:\n",
    "        print(class_.name)\n",
    "        os.chdir(class_.name)\n",
    "        classes[class_.name] = n_label\n",
    "\n",
    "        # Finer Level Subdirectories per Coarse Level\n",
    "        subclasses = os.scandir('./')\n",
    "        for subclass in subclasses:\n",
    "            os.chdir(subclass.name)\n",
    "            files = os.listdir('./')\n",
    "            for file in files:\n",
    "                image = cv2.imread(file)\n",
    "                images.append(image)\n",
    "                labels.append(n_label)\n",
    "                \n",
    "            os.chdir('../')\n",
    "\n",
    "        os.chdir('../')\n",
    "        n_label += 1\n",
    "    os.chdir('../')\n",
    "    images = np.asarray(images)\n",
    "    images = (images / 255.0).astype(np.float32)\n",
    "    labels = to_categorical(labels, n_label)\n",
    "    print(\"Images\", images.shape, \"Labels\", labels.shape, \"Classes\", classes)\n",
    "\n",
    "    # Split the processed image dataset into training and test data\n",
    "    x_train, x_test, y_train, y_test = train_test_split(images, labels, test_size=0.20, shuffle=True)\n",
    "    return x_train, x_test, y_train, y_test, classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make Finer Category Datasets\n",
    "\n",
    "This makes the by Fruit Variety datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Varieties(root):\n",
    "    ''' Generate Cascade (Finer) Level Dataset for Fruit Varieties'''\n",
    "\n",
    "    datasets = {}\n",
    "    \n",
    "    os.chdir(root)\n",
    "    fruits = os.scandir('./')\n",
    "    for fruit in fruits:\n",
    "        n_label = 0\n",
    "        images = []\n",
    "        labels = []\n",
    "        classes = {}\n",
    "        print('FRUIT', fruit.name)\n",
    "        os.chdir(fruit.name)\n",
    "        varieties = os.scandir('./')\n",
    "        for variety in varieties:\n",
    "            print('VARIETY', variety.name)\n",
    "            classes[variety.name] = n_label\n",
    "            os.chdir(variety.name)\n",
    "            files = os.listdir('./')\n",
    "            for file in files:\n",
    "                image = cv2.imread(file)\n",
    "                images.append(image)\n",
    "                labels.append(n_label)\n",
    "            os.chdir('../')\n",
    "            n_label += 1\n",
    "        images = np.asarray(images)\n",
    "        images = (images / 255.0).astype(np.float32)\n",
    "        labels = to_categorical(labels, n_label)\n",
    "        x_train, x_test, y_train, y_test = train_test_split(images, labels, test_size=0.20, shuffle=True)\n",
    "        datasets[fruit.name] = (x_train, x_test, y_train, y_test, classes)\n",
    "        os.chdir('../')\n",
    "        print(\"IMAGES\", x_train.shape, y_train.shape, \"CLASSES\", classes)\n",
    "    os.chdir('../')\n",
    "    return datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate the preprocessed Coarse Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              total        used        free      shared  buff/cache   available\n",
      "Mem:          64146        8888       45049         101       10208       54503\n",
      "Swap:         65187        5628       59559\n",
      "Avocado\n",
      "Rambutan\n",
      "Pineapple\n",
      "Kiwi\n",
      "Cantaloupe\n",
      "Apple\n",
      "Tamarillo\n",
      "Physalis\n",
      "Plum\n",
      "Pitahaya\n",
      "Guava\n",
      "Limes\n",
      "Grapefruit\n",
      "Peach\n",
      "Pomegranate\n",
      "Nectarine\n",
      "Apricot\n",
      "Banana\n",
      "Cherry\n",
      "Mulberry\n",
      "Raspberry\n",
      "Cactus Fruit\n",
      "Grape\n",
      "Mandarine\n",
      "Granadilla\n",
      "Carambula\n",
      "Passion Fruit\n",
      "Lychee\n",
      "Quince\n",
      "Maracuja\n",
      "Strawberry\n",
      "Tangelo\n",
      "Huckleberry\n",
      "Orange\n",
      "Dates\n",
      "Melon\n",
      "Pepino\n",
      "Clementine\n",
      "Papaya\n",
      "Mango\n",
      "Tomato\n",
      "Salak\n",
      "Kaki\n",
      "Pear\n",
      "Cocos\n",
      "Lemon\n",
      "Kumquats\n",
      "Images (51258, 100, 100, 3) Labels (51258, 47) Classes {'Avocado': 0, 'Rambutan': 1, 'Pineapple': 2, 'Kiwi': 3, 'Cantaloupe': 4, 'Apple': 5, 'Tamarillo': 6, 'Physalis': 7, 'Plum': 8, 'Pitahaya': 9, 'Guava': 10, 'Limes': 11, 'Grapefruit': 12, 'Peach': 13, 'Pomegranate': 14, 'Nectarine': 15, 'Apricot': 16, 'Banana': 17, 'Cherry': 18, 'Mulberry': 19, 'Raspberry': 20, 'Cactus Fruit': 21, 'Grape': 22, 'Mandarine': 23, 'Granadilla': 24, 'Carambula': 25, 'Passion Fruit': 26, 'Lychee': 27, 'Quince': 28, 'Maracuja': 29, 'Strawberry': 30, 'Tangelo': 31, 'Huckleberry': 32, 'Orange': 33, 'Dates': 34, 'Melon': 35, 'Pepino': 36, 'Clementine': 37, 'Papaya': 38, 'Mango': 39, 'Tomato': 40, 'Salak': 41, 'Kaki': 42, 'Pear': 43, 'Cocos': 44, 'Lemon': 45, 'Kumquats': 46}\n",
      "              total        used        free      shared  buff/cache   available\n",
      "Mem:          64146       16224       37722          93       10199       47175\n",
      "Swap:         65187        5628       59559\n"
     ]
    }
   ],
   "source": [
    "!free -m\n",
    "x_train, x_test, y_train, y_test, fruits_classes = Fruits('Training')\n",
    "!free -m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split Coarse Dataset (by Fruit) into Train, Validation and Test\n",
    "\n",
    "First split into train and test. Then split out 10% of train to use for validation during training.\n",
    "\n",
    "    - Train: 80%\n",
    "        - Train: 90%\n",
    "        - Validation: 10%\n",
    "    - Test : 20%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train (36905, 100, 100, 3) (36905, 47)\n",
      "val   (4101, 100, 100, 3) (4101, 47)\n",
      "test  (10252, 100, 100, 3) (10252, 47)\n",
      "              total        used        free      shared  buff/cache   available\n",
      "Mem:          64146       16255       37683         101       10207       47136\n",
      "Swap:         65187        5628       59559\n"
     ]
    }
   ],
   "source": [
    "# Split out 10% of Train to use for Validation\n",
    "pivot = int(len(x_train) * 0.9)\n",
    "x_val = x_train[pivot:]\n",
    "y_val = y_train[pivot:]\n",
    "x_train = x_train[:pivot]\n",
    "y_train = y_train[:pivot]\n",
    "\n",
    "print(\"train\", x_train.shape, y_train.shape)\n",
    "print(\"val  \", x_val.shape, y_val.shape)\n",
    "print(\"test \", x_test.shape, y_test.shape)\n",
    "!free -m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make Trainers\n",
    "\n",
    "Create the routines we will use for training.\n",
    "\n",
    "### Make Feeder\n",
    "\n",
    "Prepare the Feeder mechanism for training the neural networkm using ImageDataGenerator. \n",
    "\n",
    "Add image augmentation for:\n",
    "\n",
    "    1. Horizontal Flip\n",
    "    2. Verticial  Flip\n",
    "    3. Random Rotation +/- 30 degrees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Feeder():\n",
    "    datagen = ImageDataGenerator(horizontal_flip=True, vertical_flip=True, rotation_range=30)\n",
    "    return datagen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make Trainer\n",
    "\n",
    "Prepare a training session:\n",
    "\n",
    "    1. Epochs defaults to 10\n",
    "    2. Batch size defaults to 32\n",
    "    3. Train with validation data\n",
    "    4. Final evaluation with test data (holdout set)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Train(model, datagen, x_train, y_train, x_test, y_test, epochs=10, batch_size=32):\n",
    "    model.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size, shuffle=True),\n",
    "                    steps_per_epoch=len(x_train) / batch_size, epochs=epochs, verbose=1, validation_data=(x_test, y_test))\n",
    "    scores = model.evaluate(x_train, y_train, verbose=1)\n",
    "    print(\"Train\", scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make Model\n",
    "\n",
    "### Stem Convolutional Block (Base Model)\n",
    "\n",
    "We will use this base model as the stem convolutional block of cascading model:\n",
    "    1. The output of this model are a set of pooled feature maps.\n",
    "    2. The last layer that produces this set of pooled feature maps is referred to as the bottleneck layer.\n",
    "\n",
    "### Coarse Classifier\n",
    "\n",
    "The coarse classifier is an independent block layer for classifying the coarse level label:\n",
    "\n",
    "    1. Input is the bottleneck layer from the stem convolutional block.\n",
    "    2. Layer consists of a convolution layer and a dense layer, where the dense layer is the classifier.\n",
    "    \n",
    "### Finer Classifier\n",
    "\n",
    "The finer classifiers are a set of independent block layers for classifying the finer label. There is one finer classifier per unique coarse level label.\n",
    "\n",
    "    1. Input is the bottleneck layer from the stem convolutional block.\n",
    "    2. Layer consists of a convolution layer and a dense layer, where the dense layer is the classifier.\n",
    "    3. The finer classifer is conditionally executed based on the softmax output from the coarse classifier.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ResNet for Transfer Learning\n",
    "\n",
    "Use a prebuilt Keras model (ResNet 50). Either as:\n",
    "\n",
    "    1. Transfer Learning: The layers are pretrained with imagenet weights.\n",
    "    2. Full Training: layers are not pretrained (weights = None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ResNet(shape=(128, 128, 3), nclasses=47, optimizer='adam', weights=None):\n",
    "    base_model = ResNet50(weights=weights, include_top=False, input_shape=shape)\n",
    "\n",
    "    for i, layer in enumerate(base_model.layers):\n",
    "        # first: train only the top layers (which were randomly initialized) for Transfer Learning\n",
    "        if weights is not None:\n",
    "            layer.trainable = False\n",
    "       \n",
    "    # label the last convolutional layer in the base model as the bottleneck\n",
    "    layer.name = 'bottleneck'\n",
    "    \n",
    "    # Get the last convolutional layer of the ResNet base model\n",
    "    x = base_model.output\n",
    "    \n",
    "    # add a global spatial average pooling layer\n",
    "    x = GlobalAveragePooling2D()(x)\n",
    "    # let's add a fully-connected layer\n",
    "    #x = Dense(1024, activation='relu')(x)\n",
    "    # and a logistic layer \n",
    "    predictions = Dense(nclasses, activation='softmax')(x)\n",
    "    # this is the model we will train\n",
    "    model = Model(inputs=base_model.input, outputs=predictions)\n",
    "\n",
    "    # compile the model (should be done *after* setting layers to non-trainable)\n",
    "    model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "    model.summary()\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple ConvNet\n",
    "\n",
    "The stem convolutional block consists of a mini-VGG, which consists of:\n",
    "\n",
    "    1. A convolutional input (stem)\n",
    "    2. Three convolutional groups, each doubling the number of filers.\n",
    "    3. Each convolutional group consists of one convolutional block.\n",
    "    4. A dropout of 50% is added to the first convolutional group.\n",
    "    \n",
    "The coarse classifier consists of:\n",
    "\n",
    "    1. A 1024 none dense layer\n",
    "    2. A 47 node dense layer for classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ConvNet(shape=(128, 128, 3), nclasses=47, optimizer='adam'):\n",
    "    model = Sequential()\n",
    "    # stem convolutional group\n",
    "    model.add(Conv2D(16, (3,3), padding='same', activation='relu', input_shape=shape))\n",
    "\n",
    "    # conv block - double filters\n",
    "    model.add(Conv2D(32, (3,3), padding='same'))\n",
    "    model.add(ReLU())    \n",
    "    model.add(Dropout(0.50)) \n",
    "    model.add(MaxPooling2D((2,2)))\n",
    "    \n",
    "    # conv block - double filters\n",
    "    model.add(Conv2D(64, (3,3), padding='same'))\n",
    "    model.add(ReLU())\n",
    "    model.add(MaxPooling2D((2,2)))\n",
    "    \n",
    "    # conv block - double filters + bottleneck layer\n",
    "    model.add(Conv2D(128, (3,3), padding='same', activation='relu'))\n",
    "    model.add(MaxPooling2D((2,2), name=\"bottleneck\"))\n",
    "    \n",
    "    # dense block\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(1024, activation='relu'))\n",
    "    model.add(Dropout(0.25))\n",
    "    \n",
    "    # classifier\n",
    "    model.add(Dense(nclasses, activation='softmax'))\n",
    "    model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "    model.summary()\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start Training\n",
    "\n",
    "    1. Train the Coarse Classifier\n",
    "    2. Add Finer Classifiers\n",
    "    4. Train the Finer Classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate Coarse Model\n",
    "\n",
    "Choose between:\n",
    "\n",
    "    1. A untrained simple VGG CovNet as Stem Convolution Group, or\n",
    "    2. Pre-trained ResNet50 (imagenet weights) for Transfer Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/google/home/aferlitsch/.local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From /usr/local/google/home/aferlitsch/.local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 1024)              0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 47)                48175     \n",
      "=================================================================\n",
      "Total params: 19,021,007\n",
      "Trainable params: 19,021,007\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Select the model for the stem convolutional group (shared layers)\n",
    "stem = 'ConvNet'\n",
    "if stem == 'ConvNet':\n",
    "    model = ConvNet(shape=(100, 100, 3))\n",
    "elif stem == 'ResNet-imagenet':\n",
    "    model = ResNet(weights='imagenet', optimizer='adagrad')\n",
    "elif stem == 'ResNet':\n",
    "    model = ResNet()\n",
    "# load previously stored model\n",
    "else:\n",
    "    model = load_model('model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the Coarse Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/google/home/aferlitsch/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Epoch 1/5\n",
      "1154/1153 [==============================] - 492s 427ms/step - loss: 0.7688 - acc: 0.7665 - val_loss: 0.4038 - val_acc: 0.8800\n",
      "Epoch 2/5\n",
      "1154/1153 [==============================] - 489s 424ms/step - loss: 0.1687 - acc: 0.9431 - val_loss: 0.1588 - val_acc: 0.9537\n",
      "Epoch 3/5\n",
      "1154/1153 [==============================] - 487s 422ms/step - loss: 0.1329 - acc: 0.9529 - val_loss: 0.1037 - val_acc: 0.9659\n",
      "Epoch 4/5\n",
      "1154/1153 [==============================] - 489s 424ms/step - loss: 0.1239 - acc: 0.9589 - val_loss: 0.2824 - val_acc: 0.9056\n",
      "Epoch 5/5\n",
      "1154/1153 [==============================] - 488s 423ms/step - loss: 0.0854 - acc: 0.9683 - val_loss: 0.1708 - val_acc: 0.9361\n",
      "36905/36905 [==============================] - 90s 2ms/step\n",
      "Train [0.1527849808228305, 0.9426364991209756]\n",
      "10252/10252 [==============================] - 26s 3ms/step\n",
      "Test [0.17052910335498586, 0.935914943425673]\n"
     ]
    }
   ],
   "source": [
    "datagen = Feeder()\n",
    "Train(model, datagen, x_train, y_train, x_val, y_val, 5)\n",
    "\n",
    "scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "print(\"Test\", scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the Coarse Model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the model and weights\n",
    "model.save(\"model-coarse.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare Coarse CNN for cascade training\n",
    "\n",
    "    1. Freeze all layers\n",
    "    2. Find bottleneck layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Bottleneck(model):\n",
    "    for layer in model.layers:\n",
    "        layer.trainable = False\n",
    "        if layer.name == 'bottleneck':\n",
    "            bottleneck = layer\n",
    "\n",
    "    print(\"BOTTLENECK\", bottleneck.output.shape)\n",
    "    return bottleneck"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate the preprocessed Finer Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split Finer (by Variety) Datasets into Train, Validation and Test\n",
    "\n",
    "    1. For each fruit type, split the corresponding variety images into train, validation and test.\n",
    "    2. Save each split dataset in a dictionary, using the fruit name as the key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Converse memory by releasing training data for coarse model\n",
    "import gc\n",
    "x_train = y_train = x_val = y_val = x_test = y_test = None\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FRUIT Avocado\n",
      "VARIETY Avocado\n",
      "VARIETY Avocado ripe\n",
      "IMAGES (981, 100, 100, 3) (981, 2) CLASSES {'Avocado': 0, 'Avocado ripe': 1}\n",
      "FRUIT Rambutan\n",
      "VARIETY Rambutan\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Rambutan': 0}\n",
      "FRUIT Pineapple\n",
      "VARIETY Pineapple\n",
      "VARIETY Pineapple Mini\n",
      "IMAGES (1048, 100, 100, 3) (1048, 2) CLASSES {'Pineapple': 0, 'Pineapple Mini': 1}\n",
      "FRUIT Kiwi\n",
      "VARIETY Kiwi\n",
      "IMAGES (497, 100, 100, 3) (497, 1) CLASSES {'Kiwi': 0}\n",
      "FRUIT Cantaloupe\n",
      "VARIETY Cantaloupe 1\n",
      "VARIETY Cantaloupe 2\n",
      "IMAGES (1049, 100, 100, 3) (1049, 2) CLASSES {'Cantaloupe 1': 0, 'Cantaloupe 2': 1}\n",
      "FRUIT Apple\n",
      "VARIETY Apple Golden 2\n",
      "VARIETY Apple Red 3\n",
      "VARIETY Apple Red Yellow\n",
      "VARIETY Apple Granny Smith\n",
      "VARIETY Apple Red 2\n",
      "VARIETY Apple Braeburn\n",
      "VARIETY Apple Golden 3\n",
      "VARIETY Apple Red Delicious\n",
      "VARIETY Apple Red 1\n",
      "VARIETY Apple Golden 1\n",
      "IMAGES (5170, 100, 100, 3) (5170, 10) CLASSES {'Apple Golden 2': 0, 'Apple Red 3': 1, 'Apple Red Yellow': 2, 'Apple Granny Smith': 3, 'Apple Red 2': 4, 'Apple Braeburn': 5, 'Apple Golden 3': 6, 'Apple Red Delicious': 7, 'Apple Red 1': 8, 'Apple Golden 1': 9}\n",
      "FRUIT Tamarillo\n",
      "VARIETY Tamarillo\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Tamarillo': 0}\n",
      "FRUIT Physalis\n",
      "VARIETY Physalis\n",
      "VARIETY Physalis with Husk\n",
      "IMAGES (1049, 100, 100, 3) (1049, 2) CLASSES {'Physalis': 0, 'Physalis with Husk': 1}\n",
      "FRUIT Plum\n",
      "VARIETY Plum\n",
      "IMAGES (478, 100, 100, 3) (478, 1) CLASSES {'Plum': 0}\n",
      "FRUIT Pitahaya\n",
      "VARIETY Pitahaya Red\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Pitahaya Red': 0}\n",
      "FRUIT Guava\n",
      "VARIETY Guava\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Guava': 0}\n",
      "FRUIT Limes\n",
      "VARIETY Limes\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Limes': 0}\n",
      "FRUIT Grapefruit\n",
      "VARIETY Grapefruit White\n",
      "VARIETY Grapefruit Pink\n",
      "IMAGES (1049, 100, 100, 3) (1049, 2) CLASSES {'Grapefruit White': 0, 'Grapefruit Pink': 1}\n",
      "FRUIT Peach\n",
      "VARIETY Peach\n",
      "VARIETY Peach Flat\n",
      "IMAGES (1049, 100, 100, 3) (1049, 2) CLASSES {'Peach': 0, 'Peach Flat': 1}\n",
      "FRUIT Pomegranate\n",
      "VARIETY Pomegranate\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Pomegranate': 0}\n",
      "FRUIT Nectarine\n",
      "VARIETY Nectarine\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Nectarine': 0}\n",
      "FRUIT Apricot\n",
      "VARIETY Apricot\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Apricot': 0}\n",
      "FRUIT Banana\n",
      "VARIETY Banana\n",
      "VARIETY Banana Red\n",
      "IMAGES (1049, 100, 100, 3) (1049, 2) CLASSES {'Banana': 0, 'Banana Red': 1}\n",
      "FRUIT Cherry\n",
      "VARIETY Cherry Rainier\n",
      "VARIETY Cherry 2\n",
      "VARIETY Carambula\n",
      "VARIETY Cherry Wax Yellow\n",
      "VARIETY Cherry Wax Black\n",
      "VARIETY Cherry 1\n",
      "VARIETY Cherry Wax Red\n",
      "IMAGES (1310, 100, 100, 3) (1310, 7) CLASSES {'Cherry Rainier': 0, 'Cherry 2': 1, 'Carambula': 2, 'Cherry Wax Yellow': 3, 'Cherry Wax Black': 4, 'Cherry 1': 5, 'Cherry Wax Red': 6}\n",
      "FRUIT Mulberry\n",
      "VARIETY Mulberry\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Mulberry': 0}\n",
      "FRUIT Raspberry\n",
      "VARIETY Rambutan\n",
      "VARIETY Raspberry\n",
      "IMAGES (523, 100, 100, 3) (523, 2) CLASSES {'Rambutan': 0, 'Raspberry': 1}\n",
      "FRUIT Cactus Fruit\n",
      "VARIETY Cactus fruit\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Cactus fruit': 0}\n",
      "FRUIT Grape\n",
      "VARIETY Grape White\n",
      "VARIETY Grape Pink\n",
      "VARIETY Grape White 2\n",
      "IMAGES (1574, 100, 100, 3) (1574, 3) CLASSES {'Grape White': 0, 'Grape Pink': 1, 'Grape White 2': 2}\n",
      "FRUIT Mandarine\n",
      "VARIETY Mandarine\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Mandarine': 0}\n",
      "FRUIT Granadilla\n",
      "VARIETY Granadilla\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Granadilla': 0}\n",
      "FRUIT Carambula\n",
      "VARIETY Carambula\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Carambula': 0}\n",
      "FRUIT Passion Fruit\n",
      "VARIETY Passion Fruit\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Passion Fruit': 0}\n",
      "FRUIT Lychee\n",
      "VARIETY Lychee\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Lychee': 0}\n",
      "FRUIT Quince\n",
      "VARIETY Quince\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Quince': 0}\n",
      "FRUIT Maracuja\n",
      "VARIETY Maracuja\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Maracuja': 0}\n",
      "FRUIT Strawberry\n",
      "VARIETY Strawberry Wedge\n",
      "VARIETY Strawberry\n",
      "IMAGES (1312, 100, 100, 3) (1312, 2) CLASSES {'Strawberry Wedge': 0, 'Strawberry': 1}\n",
      "FRUIT Tangelo\n",
      "VARIETY Tangelo\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Tangelo': 0}\n",
      "FRUIT Huckleberry\n",
      "VARIETY Huckleberry\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Huckleberry': 0}\n",
      "FRUIT Orange\n",
      "VARIETY Orange\n",
      "IMAGES (511, 100, 100, 3) (511, 1) CLASSES {'Orange': 0}\n",
      "FRUIT Dates\n",
      "VARIETY Dates\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Dates': 0}\n",
      "FRUIT Melon\n",
      "VARIETY Melon Piel de Sapo\n",
      "IMAGES (787, 100, 100, 3) (787, 1) CLASSES {'Melon Piel de Sapo': 0}\n",
      "FRUIT Pepino\n",
      "VARIETY Pepino\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Pepino': 0}\n",
      "FRUIT Clementine\n",
      "VARIETY Clementine\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Clementine': 0}\n",
      "FRUIT Papaya\n",
      "VARIETY Papaya\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Papaya': 0}\n",
      "FRUIT Mango\n",
      "VARIETY Mango\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Mango': 0}\n",
      "FRUIT Tomato\n",
      "VARIETY Tomato 2\n",
      "VARIETY Tomato Cherry Red\n",
      "VARIETY Tomato Maroon\n",
      "VARIETY Tomato 1\n",
      "VARIETY Tomato 3\n",
      "VARIETY Tomato 4\n",
      "IMAGES (3723, 100, 100, 3) (3723, 6) CLASSES {'Tomato 2': 0, 'Tomato Cherry Red': 1, 'Tomato Maroon': 2, 'Tomato 1': 3, 'Tomato 3': 4, 'Tomato 4': 5}\n",
      "FRUIT Salak\n",
      "VARIETY Salak\n",
      "IMAGES (521, 100, 100, 3) (521, 1) CLASSES {'Salak': 0}\n",
      "FRUIT Kaki\n",
      "VARIETY Kaki\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Kaki': 0}\n",
      "FRUIT Pear\n",
      "VARIETY Pear Williams\n",
      "VARIETY Pear Monster\n",
      "VARIETY Pear Abate\n",
      "VARIETY Pear\n",
      "IMAGES (2099, 100, 100, 3) (2099, 4) CLASSES {'Pear Williams': 0, 'Pear Monster': 1, 'Pear Abate': 2, 'Pear': 3}\n",
      "FRUIT Cocos\n",
      "VARIETY Cocos\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Cocos': 0}\n",
      "FRUIT Lemon\n",
      "VARIETY Lemon Meyer\n",
      "VARIETY Lemon\n",
      "IMAGES (1049, 100, 100, 3) (1049, 2) CLASSES {'Lemon Meyer': 0, 'Lemon': 1}\n",
      "FRUIT Kumquats\n",
      "VARIETY Kumquats\n",
      "IMAGES (524, 100, 100, 3) (524, 1) CLASSES {'Kumquats': 0}\n",
      "              total        used        free      shared  buff/cache   available\n",
      "Mem:          64146       20753       33471          95        9921       42644\n",
      "Swap:         65187        5624       59563\n"
     ]
    }
   ],
   "source": [
    "varieties_datasets = Varieties('Training')\n",
    "for key, dataset in varieties_datasets.items():\n",
    "    \n",
    "    _x_train, _x_test, _y_train, _y_test, classes = dataset\n",
    "            \n",
    "    # Separate out 10% of train for validation\n",
    "    pivot = int(len(_x_train) * 0.9)\n",
    "    _x_val = _x_train[pivot:]\n",
    "    _y_val = _y_train[pivot:]\n",
    "    _x_train = _x_train[:pivot]\n",
    "    _y_train = _y_train[:pivot]\n",
    "    \n",
    "    # save the dataset for this fruit (key)\n",
    "    varieties_datasets[key] = { 'classes': classes, 'train': (_x_train, _y_train), 'val': (_x_val, _y_val), 'test': (_x_test, _y_test) }\n",
    "    \n",
    "!free -m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add Each Cascade (Finer) Classifier\n",
    "\n",
    "    1. Get the bottleneck layer for the coarse CNN\n",
    "    2. Add an independent finer classifier per fruit from the bottleneck layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BOTTLENECK (?, 12, 12, 128)\n",
      "KEY Avocado {'Avocado': 0, 'Avocado ripe': 1}\n",
      "KEY Rambutan {'Rambutan': 0}\n",
      "KEY Pineapple {'Pineapple': 0, 'Pineapple Mini': 1}\n",
      "KEY Kiwi {'Kiwi': 0}\n",
      "KEY Cantaloupe {'Cantaloupe 1': 0, 'Cantaloupe 2': 1}\n",
      "KEY Apple {'Apple Golden 2': 0, 'Apple Red 3': 1, 'Apple Red Yellow': 2, 'Apple Granny Smith': 3, 'Apple Red 2': 4, 'Apple Braeburn': 5, 'Apple Golden 3': 6, 'Apple Red Delicious': 7, 'Apple Red 1': 8, 'Apple Golden 1': 9}\n",
      "KEY Tamarillo {'Tamarillo': 0}\n",
      "KEY Physalis {'Physalis': 0, 'Physalis with Husk': 1}\n",
      "KEY Plum {'Plum': 0}\n",
      "KEY Pitahaya {'Pitahaya Red': 0}\n",
      "KEY Guava {'Guava': 0}\n",
      "KEY Limes {'Limes': 0}\n",
      "KEY Grapefruit {'Grapefruit White': 0, 'Grapefruit Pink': 1}\n",
      "KEY Peach {'Peach': 0, 'Peach Flat': 1}\n",
      "KEY Pomegranate {'Pomegranate': 0}\n",
      "KEY Nectarine {'Nectarine': 0}\n",
      "KEY Apricot {'Apricot': 0}\n",
      "KEY Banana {'Banana': 0, 'Banana Red': 1}\n",
      "KEY Cherry {'Cherry Rainier': 0, 'Cherry 2': 1, 'Carambula': 2, 'Cherry Wax Yellow': 3, 'Cherry Wax Black': 4, 'Cherry 1': 5, 'Cherry Wax Red': 6}\n",
      "KEY Mulberry {'Mulberry': 0}\n",
      "KEY Raspberry {'Rambutan': 0, 'Raspberry': 1}\n",
      "KEY Cactus Fruit {'Cactus fruit': 0}\n",
      "KEY Grape {'Grape White': 0, 'Grape Pink': 1, 'Grape White 2': 2}\n",
      "KEY Mandarine {'Mandarine': 0}\n",
      "KEY Granadilla {'Granadilla': 0}\n",
      "KEY Carambula {'Carambula': 0}\n",
      "KEY Passion Fruit {'Passion Fruit': 0}\n",
      "KEY Lychee {'Lychee': 0}\n",
      "KEY Quince {'Quince': 0}\n",
      "KEY Maracuja {'Maracuja': 0}\n",
      "KEY Strawberry {'Strawberry Wedge': 0, 'Strawberry': 1}\n",
      "KEY Tangelo {'Tangelo': 0}\n",
      "KEY Huckleberry {'Huckleberry': 0}\n",
      "KEY Orange {'Orange': 0}\n",
      "KEY Dates {'Dates': 0}\n",
      "KEY Melon {'Melon Piel de Sapo': 0}\n",
      "KEY Pepino {'Pepino': 0}\n",
      "KEY Clementine {'Clementine': 0}\n",
      "KEY Papaya {'Papaya': 0}\n",
      "KEY Mango {'Mango': 0}\n",
      "KEY Tomato {'Tomato 2': 0, 'Tomato Cherry Red': 1, 'Tomato Maroon': 2, 'Tomato 1': 3, 'Tomato 3': 4, 'Tomato 4': 5}\n",
      "KEY Salak {'Salak': 0}\n",
      "KEY Kaki {'Kaki': 0}\n",
      "KEY Pear {'Pear Williams': 0, 'Pear Monster': 1, 'Pear Abate': 2, 'Pear': 3}\n",
      "KEY Cocos {'Cocos': 0}\n",
      "KEY Lemon {'Lemon Meyer': 0, 'Lemon': 1}\n",
      "KEY Kumquats {'Kumquats': 0}\n"
     ]
    }
   ],
   "source": [
    "bottleneck = Bottleneck(model)\n",
    "cascades = []\n",
    "for key, val in varieties_datasets.items():\n",
    "    classes = val['classes']\n",
    "    print(\"KEY\", key, classes)\n",
    "    # if only one subclassifier, then skip (i.e., coarse == finer)\n",
    "    if len(classes) == 1:\n",
    "        continue\n",
    "    x = layers.Conv2D(128, (3,3), padding='same', activation='relu')(bottleneck.output)\n",
    "    x = BatchNormalization()(x)\n",
    "    x = MaxPooling2D((2,2))(x)\n",
    "    \n",
    "    x = layers.Flatten()(bottleneck.output)\n",
    "    x = layers.Dense(1024, activation='relu')(x)\n",
    "    x = layers.Dense(len(classes), activation='softmax', name=key.replace(' ', ''))(x)\n",
    "    cascades.append(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compile each finer classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Avocado (Dense)              (None, 2)                 2050      \n",
      "=================================================================\n",
      "Total params: 18,974,882\n",
      "Trainable params: 18,877,442\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_3 (Flatten)          (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Pineapple (Dense)            (None, 2)                 2050      \n",
      "=================================================================\n",
      "Total params: 18,974,882\n",
      "Trainable params: 18,877,442\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_4 (Flatten)          (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Cantaloupe (Dense)           (None, 2)                 2050      \n",
      "=================================================================\n",
      "Total params: 18,974,882\n",
      "Trainable params: 18,877,442\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_5 (Flatten)          (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Apple (Dense)                (None, 10)                10250     \n",
      "=================================================================\n",
      "Total params: 18,983,082\n",
      "Trainable params: 18,885,642\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_6 (Flatten)          (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Physalis (Dense)             (None, 2)                 2050      \n",
      "=================================================================\n",
      "Total params: 18,974,882\n",
      "Trainable params: 18,877,442\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_7 (Flatten)          (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Grapefruit (Dense)           (None, 2)                 2050      \n",
      "=================================================================\n",
      "Total params: 18,974,882\n",
      "Trainable params: 18,877,442\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_8 (Flatten)          (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Peach (Dense)                (None, 2)                 2050      \n",
      "=================================================================\n",
      "Total params: 18,974,882\n",
      "Trainable params: 18,877,442\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_9 (Flatten)          (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Banana (Dense)               (None, 2)                 2050      \n",
      "=================================================================\n",
      "Total params: 18,974,882\n",
      "Trainable params: 18,877,442\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_10 (Flatten)         (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_11 (Dense)             (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Cherry (Dense)               (None, 7)                 7175      \n",
      "=================================================================\n",
      "Total params: 18,980,007\n",
      "Trainable params: 18,882,567\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_11 (Flatten)         (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_12 (Dense)             (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Raspberry (Dense)            (None, 2)                 2050      \n",
      "=================================================================\n",
      "Total params: 18,974,882\n",
      "Trainable params: 18,877,442\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_12 (Flatten)         (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_13 (Dense)             (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Grape (Dense)                (None, 3)                 3075      \n",
      "=================================================================\n",
      "Total params: 18,975,907\n",
      "Trainable params: 18,878,467\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_13 (Flatten)         (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_14 (Dense)             (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Strawberry (Dense)           (None, 2)                 2050      \n",
      "=================================================================\n",
      "Total params: 18,974,882\n",
      "Trainable params: 18,877,442\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_14 (Flatten)         (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_15 (Dense)             (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Tomato (Dense)               (None, 6)                 6150      \n",
      "=================================================================\n",
      "Total params: 18,978,982\n",
      "Trainable params: 18,881,542\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_15 (Flatten)         (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_16 (Dense)             (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Pear (Dense)                 (None, 4)                 4100      \n",
      "=================================================================\n",
      "Total params: 18,976,932\n",
      "Trainable params: 18,879,492\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1_input (InputLayer)  (None, 100, 100, 3)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 100, 100, 16)      448       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 100, 100, 32)      4640      \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100, 100, 32)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 50, 50, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 50, 50, 64)        18496     \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 50, 50, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 25, 25, 128)       73856     \n",
      "_________________________________________________________________\n",
      "bottleneck (MaxPooling2D)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "flatten_16 (Flatten)         (None, 18432)             0         \n",
      "_________________________________________________________________\n",
      "dense_17 (Dense)             (None, 1024)              18875392  \n",
      "_________________________________________________________________\n",
      "Lemon (Dense)                (None, 2)                 2050      \n",
      "=================================================================\n",
      "Total params: 18,974,882\n",
      "Trainable params: 18,877,442\n",
      "Non-trainable params: 97,440\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "classifiers = []\n",
    "for cascade in cascades:\n",
    "    _model = Model(model.input, cascade)\n",
    "    _model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    _model.summary()\n",
    "    classifiers.append(_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the finer classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<keras.layers.core.Dense object at 0x7f622995a9b0> Avocado\n",
      "Epoch 1/5\n",
      "28/27 [==============================] - 8s 274ms/step - loss: 0.0588 - acc: 0.9620 - val_loss: 1.1626e-04 - val_acc: 1.0000\n",
      "Epoch 2/5\n",
      "28/27 [==============================] - 5s 196ms/step - loss: 3.6445e-06 - acc: 1.0000 - val_loss: 4.3129e-05 - val_acc: 1.0000\n",
      "Epoch 3/5\n",
      "28/27 [==============================] - 5s 194ms/step - loss: 8.5374e-07 - acc: 1.0000 - val_loss: 4.0188e-05 - val_acc: 1.0000\n",
      "Epoch 4/5\n",
      "28/27 [==============================] - 5s 190ms/step - loss: 8.1645e-07 - acc: 1.0000 - val_loss: 3.9653e-05 - val_acc: 1.0000\n",
      "Epoch 5/5\n",
      "28/27 [==============================] - 5s 182ms/step - loss: 6.9528e-07 - acc: 1.0000 - val_loss: 3.9208e-05 - val_acc: 1.0000\n",
      "882/882 [==============================] - 2s 2ms/step\n",
      "Train [3.1851555930955815e-05, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f62297766a0> Pineapple\n",
      "Epoch 1/5\n",
      "30/29 [==============================] - 6s 216ms/step - loss: 0.0656 - acc: 0.9666 - val_loss: 8.1402e-04 - val_acc: 1.0000\n",
      "Epoch 2/5\n",
      "30/29 [==============================] - 5s 181ms/step - loss: 1.3917e-05 - acc: 1.0000 - val_loss: 0.0022 - val_acc: 1.0000\n",
      "Epoch 3/5\n",
      "30/29 [==============================] - 6s 185ms/step - loss: 5.6931e-06 - acc: 1.0000 - val_loss: 0.0031 - val_acc: 1.0000\n",
      "Epoch 4/5\n",
      "30/29 [==============================] - 6s 185ms/step - loss: 3.4368e-06 - acc: 1.0000 - val_loss: 0.0033 - val_acc: 1.0000\n",
      "Epoch 5/5\n",
      "30/29 [==============================] - 6s 185ms/step - loss: 4.9285e-06 - acc: 1.0000 - val_loss: 0.0029 - val_acc: 1.0000\n",
      "943/943 [==============================] - 2s 2ms/step\n",
      "Train [0.0017961749410965852, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f62296131d0> Cantaloupe\n",
      "Epoch 1/5\n",
      "30/29 [==============================] - 6s 215ms/step - loss: 0.0263 - acc: 0.9719 - val_loss: 1.9471e-07 - val_acc: 1.0000\n",
      "Epoch 2/5\n",
      "30/29 [==============================] - 6s 186ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 1.5270e-07 - val_acc: 1.0000\n",
      "Epoch 3/5\n",
      "30/29 [==============================] - 6s 191ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 1.5100e-07 - val_acc: 1.0000\n",
      "Epoch 4/5\n",
      "30/29 [==============================] - 6s 192ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 1.5043e-07 - val_acc: 1.0000\n",
      "Epoch 5/5\n",
      "30/29 [==============================] - 6s 191ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 1.5043e-07 - val_acc: 1.0000\n",
      "944/944 [==============================] - 2s 3ms/step\n",
      "Train [1.5387344834087314e-07, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f6229493ac8> Apple\n",
      "Epoch 1/5\n",
      "146/145 [==============================] - 28s 195ms/step - loss: 0.2106 - acc: 0.9238 - val_loss: 0.2795 - val_acc: 0.8762\n",
      "Epoch 2/5\n",
      "146/145 [==============================] - 27s 186ms/step - loss: 0.0223 - acc: 0.9940 - val_loss: 0.2492 - val_acc: 0.8878\n",
      "Epoch 3/5\n",
      "146/145 [==============================] - 27s 184ms/step - loss: 0.0170 - acc: 0.9955 - val_loss: 0.2986 - val_acc: 0.8607\n",
      "Epoch 4/5\n",
      "146/145 [==============================] - 27s 186ms/step - loss: 0.0125 - acc: 0.9955 - val_loss: 0.4114 - val_acc: 0.8162\n",
      "Epoch 5/5\n",
      "146/145 [==============================] - 27s 184ms/step - loss: 0.0033 - acc: 0.9994 - val_loss: 0.4460 - val_acc: 0.8240\n",
      "4653/4653 [==============================] - 11s 2ms/step\n",
      "Train [0.517334802097437, 0.7986245433438242]\n",
      "<keras.layers.core.Dense object at 0x7f62292ad828> Physalis\n",
      "Epoch 1/5\n",
      "30/29 [==============================] - 7s 218ms/step - loss: 0.0232 - acc: 0.9844 - val_loss: 5.0205e-06 - val_acc: 1.0000\n",
      "Epoch 2/5\n",
      "30/29 [==============================] - 5s 183ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 3.2874e-06 - val_acc: 1.0000\n",
      "Epoch 3/5\n",
      "30/29 [==============================] - 6s 186ms/step - loss: 1.2008e-07 - acc: 1.0000 - val_loss: 3.2096e-06 - val_acc: 1.0000\n",
      "Epoch 4/5\n",
      "30/29 [==============================] - 6s 186ms/step - loss: 1.1958e-07 - acc: 1.0000 - val_loss: 3.2028e-06 - val_acc: 1.0000\n",
      "Epoch 5/5\n",
      "30/29 [==============================] - 6s 186ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 3.2022e-06 - val_acc: 1.0000\n",
      "944/944 [==============================] - 2s 2ms/step\n",
      "Train [2.059461342284167e-05, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f60ae144358> Grapefruit\n",
      "Epoch 1/5\n",
      "30/29 [==============================] - 6s 216ms/step - loss: 0.0242 - acc: 0.9812 - val_loss: 2.2119e-05 - val_acc: 1.0000\n",
      "Epoch 2/5\n",
      "30/29 [==============================] - 5s 182ms/step - loss: 1.2871e-07 - acc: 1.0000 - val_loss: 1.5222e-05 - val_acc: 1.0000\n",
      "Epoch 3/5\n",
      "30/29 [==============================] - 6s 189ms/step - loss: 1.2629e-07 - acc: 1.0000 - val_loss: 1.4822e-05 - val_acc: 1.0000\n",
      "Epoch 4/5\n",
      "30/29 [==============================] - 5s 183ms/step - loss: 1.3039e-07 - acc: 1.0000 - val_loss: 1.4711e-05 - val_acc: 1.0000\n",
      "Epoch 5/5\n",
      "30/29 [==============================] - 6s 186ms/step - loss: 1.2753e-07 - acc: 1.0000 - val_loss: 1.4579e-05 - val_acc: 1.0000\n",
      "944/944 [==============================] - 2s 3ms/step\n",
      "Train [2.3263756424890065e-05, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f60adfcbfd0> Peach\n",
      "Epoch 1/5\n",
      "30/29 [==============================] - 7s 222ms/step - loss: 0.0416 - acc: 0.9812 - val_loss: 0.0040 - val_acc: 1.0000\n",
      "Epoch 2/5\n",
      "30/29 [==============================] - 6s 189ms/step - loss: 0.0012 - acc: 0.9990 - val_loss: 0.0535 - val_acc: 0.9810\n",
      "Epoch 3/5\n",
      "30/29 [==============================] - 6s 185ms/step - loss: 2.2926e-04 - acc: 1.0000 - val_loss: 0.0015 - val_acc: 1.0000\n",
      "Epoch 4/5\n",
      "30/29 [==============================] - 6s 188ms/step - loss: 6.4975e-07 - acc: 1.0000 - val_loss: 0.0011 - val_acc: 1.0000\n",
      "Epoch 5/5\n",
      "30/29 [==============================] - 6s 188ms/step - loss: 4.1982e-07 - acc: 1.0000 - val_loss: 0.0011 - val_acc: 1.0000\n",
      "944/944 [==============================] - 2s 2ms/step\n",
      "Train [0.0010120372087424013, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f60adde6f28> Banana\n",
      "Epoch 1/5\n",
      "30/29 [==============================] - 7s 219ms/step - loss: 0.0470 - acc: 0.9719 - val_loss: 0.0018 - val_acc: 1.0000\n",
      "Epoch 2/5\n",
      "30/29 [==============================] - 6s 185ms/step - loss: 8.4761e-06 - acc: 1.0000 - val_loss: 7.3545e-04 - val_acc: 1.0000\n",
      "Epoch 3/5\n",
      "30/29 [==============================] - 5s 181ms/step - loss: 1.0674e-05 - acc: 1.0000 - val_loss: 6.4712e-04 - val_acc: 1.0000\n",
      "Epoch 4/5\n",
      "30/29 [==============================] - 6s 185ms/step - loss: 5.5472e-06 - acc: 1.0000 - val_loss: 6.1552e-04 - val_acc: 1.0000\n",
      "Epoch 5/5\n",
      "30/29 [==============================] - 6s 187ms/step - loss: 4.6980e-06 - acc: 1.0000 - val_loss: 5.8839e-04 - val_acc: 1.0000\n",
      "944/944 [==============================] - 2s 2ms/step\n",
      "Train [0.0003026937695922551, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f60adc7aa20> Cherry\n",
      "Epoch 1/5\n",
      "37/36 [==============================] - 8s 216ms/step - loss: 0.2950 - acc: 0.8860 - val_loss: 0.1023 - val_acc: 0.9618\n",
      "Epoch 2/5\n",
      "37/36 [==============================] - 7s 184ms/step - loss: 0.1187 - acc: 0.9568 - val_loss: 0.0552 - val_acc: 0.9847\n",
      "Epoch 3/5\n",
      "37/36 [==============================] - 7s 188ms/step - loss: 0.1273 - acc: 0.9489 - val_loss: 0.0889 - val_acc: 0.9695\n",
      "Epoch 4/5\n",
      "37/36 [==============================] - 7s 188ms/step - loss: 0.0855 - acc: 0.9634 - val_loss: 0.0581 - val_acc: 0.9924\n",
      "Epoch 5/5\n",
      "37/36 [==============================] - 7s 192ms/step - loss: 0.0918 - acc: 0.9583 - val_loss: 0.0609 - val_acc: 0.9847\n",
      "1179/1179 [==============================] - 3s 3ms/step\n",
      "Train [0.09612723731888463, 0.9609838851535594]\n",
      "<keras.layers.core.Dense object at 0x7f60a9e78550> Raspberry\n",
      "Epoch 1/5\n",
      "15/14 [==============================] - 4s 257ms/step - loss: 0.0347 - acc: 0.9937 - val_loss: 6.7927e-07 - val_acc: 1.0000\n",
      "Epoch 2/5\n",
      "15/14 [==============================] - 3s 187ms/step - loss: 1.2070e-07 - acc: 1.0000 - val_loss: 1.3833e-07 - val_acc: 1.0000\n",
      "Epoch 3/5\n",
      "15/14 [==============================] - 3s 182ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 1.2483e-07 - val_acc: 1.0000\n",
      "Epoch 4/5\n",
      "15/14 [==============================] - 3s 187ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 1.2146e-07 - val_acc: 1.0000\n",
      "Epoch 5/5\n",
      "15/14 [==============================] - 3s 180ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 1.2146e-07 - val_acc: 1.0000\n",
      "470/470 [==============================] - 1s 2ms/step\n",
      "Train [1.2440884366948555e-07, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f60a9cf3b70> Grape\n",
      "Epoch 1/5\n",
      "45/44 [==============================] - 10s 212ms/step - loss: 0.0321 - acc: 0.9847 - val_loss: 0.4649 - val_acc: 0.7658\n",
      "Epoch 2/5\n",
      "45/44 [==============================] - 8s 184ms/step - loss: 1.2833e-05 - acc: 1.0000 - val_loss: 0.6064 - val_acc: 0.7278\n",
      "Epoch 3/5\n",
      "45/44 [==============================] - 9s 193ms/step - loss: 7.4392e-06 - acc: 1.0000 - val_loss: 0.7033 - val_acc: 0.6962\n",
      "Epoch 4/5\n",
      "45/44 [==============================] - 8s 189ms/step - loss: 6.0521e-06 - acc: 1.0000 - val_loss: 0.6891 - val_acc: 0.7025\n",
      "Epoch 5/5\n",
      "45/44 [==============================] - 8s 185ms/step - loss: 5.5889e-06 - acc: 1.0000 - val_loss: 0.7612 - val_acc: 0.6835\n",
      "1416/1416 [==============================] - 3s 2ms/step\n",
      "Train [0.7892591367333622, 0.702683615819209]\n",
      "<keras.layers.core.Dense object at 0x7f60a9b935c0> Strawberry\n",
      "Epoch 1/5\n",
      "37/36 [==============================] - 8s 218ms/step - loss: 0.0207 - acc: 0.9890 - val_loss: 2.8583e-07 - val_acc: 1.0000\n",
      "Epoch 2/5\n",
      "37/36 [==============================] - 7s 191ms/step - loss: 1.9574e-07 - acc: 1.0000 - val_loss: 2.3029e-07 - val_acc: 1.0000\n",
      "Epoch 3/5\n",
      "37/36 [==============================] - 7s 185ms/step - loss: 2.5435e-07 - acc: 1.0000 - val_loss: 2.3029e-07 - val_acc: 1.0000\n",
      "Epoch 4/5\n",
      "37/36 [==============================] - 7s 185ms/step - loss: 7.2514e-07 - acc: 1.0000 - val_loss: 2.3661e-07 - val_acc: 1.0000\n",
      "Epoch 5/5\n",
      "37/36 [==============================] - 7s 186ms/step - loss: 3.0224e-07 - acc: 1.0000 - val_loss: 2.3977e-07 - val_acc: 1.0000\n",
      "1180/1180 [==============================] - 3s 2ms/step\n",
      "Train [1.0047292608140011e-06, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f60a99b9550> Tomato\n",
      "Epoch 1/5\n",
      "105/104 [==============================] - 21s 197ms/step - loss: 0.1086 - acc: 0.9554 - val_loss: 0.0535 - val_acc: 0.9973\n",
      "Epoch 2/5\n",
      "105/104 [==============================] - 20s 186ms/step - loss: 0.0180 - acc: 0.9946 - val_loss: 0.0643 - val_acc: 0.9946\n",
      "Epoch 3/5\n",
      "105/104 [==============================] - 20s 186ms/step - loss: 0.0029 - acc: 0.9988 - val_loss: 0.0451 - val_acc: 0.9946\n",
      "Epoch 4/5\n",
      "105/104 [==============================] - 20s 186ms/step - loss: 0.0150 - acc: 0.9955 - val_loss: 0.0517 - val_acc: 0.9812\n",
      "Epoch 5/5\n",
      "105/104 [==============================] - 20s 188ms/step - loss: 0.0070 - acc: 0.9976 - val_loss: 0.0168 - val_acc: 1.0000\n",
      "3350/3350 [==============================] - 8s 2ms/step\n",
      "Train [0.014588202945888043, 0.9988059701492538]\n",
      "<keras.layers.core.Dense object at 0x7f60a9830390> Pear\n",
      "Epoch 1/5\n",
      "60/59 [==============================] - 14s 236ms/step - loss: 0.0529 - acc: 0.9755 - val_loss: 0.0268 - val_acc: 1.0000\n",
      "Epoch 2/5\n",
      "60/59 [==============================] - 13s 212ms/step - loss: 7.2313e-05 - acc: 1.0000 - val_loss: 0.0057 - val_acc: 1.0000\n",
      "Epoch 3/5\n",
      "60/59 [==============================] - 13s 212ms/step - loss: 3.2620e-05 - acc: 1.0000 - val_loss: 0.0061 - val_acc: 1.0000\n",
      "Epoch 4/5\n",
      "60/59 [==============================] - 13s 213ms/step - loss: 1.8335e-05 - acc: 1.0000 - val_loss: 0.0059 - val_acc: 1.0000\n",
      "Epoch 5/5\n",
      "60/59 [==============================] - 13s 211ms/step - loss: 2.7962e-05 - acc: 1.0000 - val_loss: 0.0064 - val_acc: 1.0000\n",
      "1889/1889 [==============================] - 4s 2ms/step\n",
      "Train [0.00811844131755981, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f60a96ccb00> Lemon\n",
      "Epoch 1/5\n",
      "30/29 [==============================] - 7s 224ms/step - loss: 0.0224 - acc: 0.9865 - val_loss: 2.0606e-07 - val_acc: 1.0000\n",
      "Epoch 2/5\n",
      "30/29 [==============================] - 5s 182ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 1.6576e-07 - val_acc: 1.0000\n",
      "Epoch 3/5\n",
      "30/29 [==============================] - 5s 182ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 1.6462e-07 - val_acc: 1.0000\n",
      "Epoch 4/5\n",
      "30/29 [==============================] - 5s 182ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 1.6462e-07 - val_acc: 1.0000\n",
      "Epoch 5/5\n",
      "30/29 [==============================] - 5s 183ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 1.6462e-07 - val_acc: 1.0000\n",
      "944/944 [==============================] - 2s 2ms/step\n",
      "Train [1.9251557413384544e-07, 1.0]\n"
     ]
    }
   ],
   "source": [
    "for classifier in classifiers:\n",
    "    # get the output layer for this subclassifier\n",
    "    last = classifier.layers[len(classifier.layers)-1]\n",
    "    print(last, last.name)\n",
    "    \n",
    "    # find the corresponding variety dataset\n",
    "    for key, dataset in varieties_datasets.items():\n",
    "        if key == last.name:\n",
    "            x_train, y_train = dataset['train']\n",
    "            x_val, y_val     = dataset['val']\n",
    "\n",
    "            datagen = Feeder()\n",
    "            Train(classifier, datagen, x_train, y_train, x_val, y_val, 5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate the Model\n",
    "\n",
    "    1. Evaluate the Model for each finer classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<keras.layers.core.Dense object at 0x7f622995a9b0> Avocado\n",
      "246/246 [==============================] - 1s 3ms/step\n",
      "Test [3.334964506992525e-05, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f62297766a0> Pineapple\n",
      "263/263 [==============================] - 1s 2ms/step\n",
      "Test [0.0016150613485378684, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f62296131d0> Cantaloupe\n",
      "263/263 [==============================] - 1s 3ms/step\n",
      "Test [1.6362951567596597e-07, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f6229493ac8> Apple\n",
      "1293/1293 [==============================] - 3s 2ms/step\n",
      "Test [0.5182560322848015, 0.7973704563492688]\n",
      "<keras.layers.core.Dense object at 0x7f62292ad828> Physalis\n",
      "263/263 [==============================] - 1s 2ms/step\n",
      "Test [2.5120300804652986e-06, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f60ae144358> Grapefruit\n",
      "263/263 [==============================] - 1s 2ms/step\n",
      "Test [2.3878306024448906e-05, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f60adfcbfd0> Peach\n",
      "263/263 [==============================] - 1s 3ms/step\n",
      "Test [0.0013185769895856542, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f60adde6f28> Banana\n",
      "263/263 [==============================] - 1s 2ms/step\n",
      "Test [0.0003372718393437462, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f60adc7aa20> Cherry\n",
      "328/328 [==============================] - 1s 3ms/step\n",
      "Test [0.09346718377456432, 0.9512195121951219]\n",
      "<keras.layers.core.Dense object at 0x7f60a9e78550> Raspberry\n",
      "131/131 [==============================] - 0s 2ms/step\n",
      "Test [1.1966428716458243e-07, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f60a9cf3b70> Grape\n",
      "394/394 [==============================] - 1s 2ms/step\n",
      "Test [0.7861503450398518, 0.710659898779719]\n",
      "<keras.layers.core.Dense object at 0x7f60a9b935c0> Strawberry\n",
      "328/328 [==============================] - 1s 2ms/step\n",
      "Test [0.001072080738674131, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f60a99b9550> Tomato\n",
      "931/931 [==============================] - 2s 2ms/step\n",
      "Test [0.017835404435782888, 0.9967776584317938]\n",
      "<keras.layers.core.Dense object at 0x7f60a9830390> Pear\n",
      "525/525 [==============================] - 1s 2ms/step\n",
      "Test [0.008367355272528671, 1.0]\n",
      "<keras.layers.core.Dense object at 0x7f60a96ccb00> Lemon\n",
      "263/263 [==============================] - 1s 3ms/step\n",
      "Test [1.5025812138251395e-07, 1.0]\n"
     ]
    }
   ],
   "source": [
    "for classifier in classifiers:\n",
    "    # get the output layer for this subclassifier\n",
    "    last = classifier.layers[len(classifier.layers)-1]\n",
    "    print(last, last.name)\n",
    "    \n",
    "    # find the corresponding variety dataset\n",
    "    for key, dataset in varieties_datasets.items():\n",
    "        if key == last.name:\n",
    "            x_test, y_test = dataset['test']\n",
    "            scores = classifier.evaluate(x_test, y_test, verbose=1)\n",
    "            print(\"Test\", scores)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the Finer Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 0\n",
    "for classifier in classifiers:\n",
    "    classifier.save('model-finer-' + str(n) + '.h5')\n",
    "    n += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's do some cascading predictions\n",
    "\n",
    "We will take one random selected image per type of fruit, and:\n",
    "\n",
    "    1. Run the image through the coarse classifier (by fruit).\n",
    "    2. Based on the predicted output, select the corresponding finer classifier (by variety).\n",
    "    3. Run the image through the corresponding finer classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Yhat 0 Coarse Prediction Avocado = Avocado\n",
      "Yhat 1 Finer  Prediction Avocado ripe = Avocado ripe\n",
      "Yhat 20 Coarse Prediction Rambutan = Raspberry\n",
      "Yhat 2 Coarse Prediction Pineapple = Pineapple\n",
      "Yhat 0 Finer  Prediction Pineapple = Pineapple\n",
      "Yhat 3 Coarse Prediction Kiwi = Kiwi\n",
      "No Finer Classifier\n",
      "Yhat 4 Coarse Prediction Cantaloupe = Cantaloupe\n",
      "Yhat 1 Finer  Prediction Cantaloupe 2 = Cantaloupe 2\n",
      "Yhat 5 Coarse Prediction Apple = Apple\n",
      "Yhat 4 Finer  Prediction Apple Red 2 = Apple Red 2\n",
      "Yhat 6 Coarse Prediction Tamarillo = Tamarillo\n",
      "No Finer Classifier\n",
      "Yhat 7 Coarse Prediction Physalis = Physalis\n",
      "Yhat 1 Finer  Prediction Physalis with Husk = Physalis with Husk\n",
      "Yhat 8 Coarse Prediction Plum = Plum\n",
      "No Finer Classifier\n",
      "Yhat 9 Coarse Prediction Pitahaya = Pitahaya\n",
      "No Finer Classifier\n",
      "Yhat 10 Coarse Prediction Guava = Guava\n",
      "No Finer Classifier\n",
      "Yhat 11 Coarse Prediction Limes = Limes\n",
      "No Finer Classifier\n",
      "Yhat 12 Coarse Prediction Grapefruit = Grapefruit\n",
      "Yhat 1 Finer  Prediction Grapefruit Pink = Grapefruit Pink\n",
      "Yhat 13 Coarse Prediction Peach = Peach\n",
      "Yhat 0 Finer  Prediction Peach = Peach\n",
      "Yhat 14 Coarse Prediction Pomegranate = Pomegranate\n",
      "No Finer Classifier\n",
      "Yhat 15 Coarse Prediction Nectarine = Nectarine\n",
      "No Finer Classifier\n",
      "Yhat 16 Coarse Prediction Apricot = Apricot\n",
      "No Finer Classifier\n",
      "Yhat 17 Coarse Prediction Banana = Banana\n",
      "Yhat 0 Finer  Prediction Banana = Banana\n",
      "Yhat 18 Coarse Prediction Cherry = Cherry\n",
      "Yhat 2 Finer  Prediction Carambula = Carambula\n",
      "Yhat 19 Coarse Prediction Mulberry = Mulberry\n",
      "No Finer Classifier\n",
      "Yhat 20 Coarse Prediction Raspberry = Raspberry\n",
      "Yhat 1 Finer  Prediction Raspberry = Raspberry\n",
      "Yhat 35 Coarse Prediction Cactus Fruit = Melon\n",
      "Yhat 22 Coarse Prediction Grape = Grape\n",
      "Yhat 2 Finer  Prediction Grape White 2 = Grape White 2\n",
      "Yhat 23 Coarse Prediction Mandarine = Mandarine\n",
      "No Finer Classifier\n",
      "Yhat 24 Coarse Prediction Granadilla = Granadilla\n",
      "No Finer Classifier\n",
      "Yhat 18 Coarse Prediction Carambula = Cherry\n",
      "Yhat 26 Coarse Prediction Passion Fruit = Passion Fruit\n",
      "No Finer Classifier\n",
      "Yhat 27 Coarse Prediction Lychee = Lychee\n",
      "No Finer Classifier\n",
      "Yhat 28 Coarse Prediction Quince = Quince\n",
      "No Finer Classifier\n",
      "Yhat 29 Coarse Prediction Maracuja = Maracuja\n",
      "No Finer Classifier\n",
      "Yhat 30 Coarse Prediction Strawberry = Strawberry\n",
      "Yhat 0 Finer  Prediction Strawberry Wedge = Strawberry Wedge\n",
      "Yhat 31 Coarse Prediction Tangelo = Tangelo\n",
      "No Finer Classifier\n",
      "Yhat 32 Coarse Prediction Huckleberry = Huckleberry\n",
      "No Finer Classifier\n",
      "Yhat 33 Coarse Prediction Orange = Orange\n",
      "No Finer Classifier\n",
      "Yhat 34 Coarse Prediction Dates = Dates\n",
      "No Finer Classifier\n",
      "Yhat 35 Coarse Prediction Melon = Melon\n",
      "No Finer Classifier\n",
      "Yhat 36 Coarse Prediction Pepino = Pepino\n",
      "No Finer Classifier\n",
      "Yhat 37 Coarse Prediction Clementine = Clementine\n",
      "No Finer Classifier\n",
      "Yhat 38 Coarse Prediction Papaya = Papaya\n",
      "No Finer Classifier\n",
      "Yhat 39 Coarse Prediction Mango = Mango\n",
      "No Finer Classifier\n",
      "Yhat 40 Coarse Prediction Tomato = Tomato\n",
      "Yhat 1 Finer  Prediction Tomato Cherry Red = Tomato Cherry Red\n",
      "Yhat 41 Coarse Prediction Salak = Salak\n",
      "No Finer Classifier\n",
      "Yhat 42 Coarse Prediction Kaki = Kaki\n",
      "No Finer Classifier\n",
      "Yhat 43 Coarse Prediction Pear = Pear\n",
      "Yhat 0 Finer  Prediction Pear Williams = Pear Williams\n",
      "Yhat 44 Coarse Prediction Cocos = Cocos\n",
      "No Finer Classifier\n",
      "Yhat 45 Coarse Prediction Lemon = Lemon\n",
      "Yhat 0 Finer  Prediction Lemon Meyer = Lemon Meyer\n",
      "Yhat 46 Coarse Prediction Kumquats = Kumquats\n",
      "No Finer Classifier\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "# Let's make a prediction for each type of fruit\n",
    "for key, dataset in varieties_datasets.items():\n",
    "    \n",
    "    # Get the variety test data for this type of fruit\n",
    "    x_test, y_test = dataset['test']\n",
    "    \n",
    "    # pick a random image in the variety datast\n",
    "    index = random.randint(0, len(x_test))\n",
    "    \n",
    "    # use the coarse model to predict the type of fruit\n",
    "    yhat = np.argmax( model.predict(x_test[index:index+1]) )\n",
    "    \n",
    "    # let's find the class name (type of fruit) for this predicted label\n",
    "    for fruit, label in fruits_classes.items():\n",
    "        if label == yhat:\n",
    "            break\n",
    "    \n",
    "    print(\"Yhat\", yhat, \"Coarse Prediction\", key, \"=\", fruit)\n",
    "    \n",
    "    # Prediction was correct\n",
    "    if key == fruit:\n",
    "        if len(dataset['classes']) == 1:\n",
    "            print(\"No Finer Classifier\")\n",
    "            continue\n",
    "            \n",
    "        # find the corresponding finer classifier for this type of fruit\n",
    "        for classifier in classifiers:\n",
    "            # get the output layer for this subclassifier\n",
    "            last = classifier.layers[len(classifier.layers)-1]\n",
    "            if last.name == fruit:\n",
    "                # use the finer model to predict the variety of this type of fruit\n",
    "                yhat = np.argmax(classifier.predict(x_test[index:index+1]))\n",
    "                for variety, value in dataset['classes'].items():\n",
    "                    if value == np.argmax(y_test[index]):\n",
    "                        break\n",
    "                for yhat_variety, value in dataset['classes'].items():\n",
    "                    if value == yhat:\n",
    "                        break\n",
    "                print(\"Yhat\", yhat, \"Finer  Prediction\", variety, \"=\", yhat_variety)\n",
    "                break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### End of Notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#     extractfeatures = Model(input=model.input, output=model.get_layer('bottleneck').output)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
